Tool position and identification indicator displayed in a boundary area of a computer display screen

ABSTRACT

An endoscope captures images of a surgical site for display in a viewing area of a monitor. When a tool is outside the viewing area, a GUI indicates the position of the tool by positioning a symbol in a boundary area around the viewing area so as to indicate the tool position. The distance of the out-of-view tool from the viewing area may be indicated by the size, color, brightness, or blinking or oscillation frequency of the symbol. A distance number may also be displayed on the symbol. The orientation of the shaft or end effector of the tool may be indicated by an orientation indicator superimposed over the symbol, or by the orientation of the symbol itself. When the tool is inside the viewing area, but occluded by an object, the GUI superimposes a ghost tool at its current position and orientation over the occluding object.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a divisional of application Ser. No. 11/478,531(filed Jun. 29, 2006), which is incorporated herein by reference.

FIELD OF THE INVENTION

The present invention generally relates to robotic surgical systems andin particular, to a tool position and identification indicator displayedin a boundary area of a computer display screen.

BACKGROUND

Robotic surgical systems such as those used in performing minimallyinvasive surgical procedures offer many benefits over traditional opensurgery techniques, including less pain, shorter hospital stays, quickerreturn to normal activities, minimal scarring, reduced recovery time,and less injury to tissue. Consequently, demand for minimally invasivesurgery using robotic surgical systems is strong and growing.

One example of a robotic surgical system is the da Vinci® SurgicalSystem from Intuitive Surgical, Inc., of Sunnyvale, Calif. The da Vinci®system includes a surgeon's console, a patient-side cart, a highperformance 3-D vision system, and Intuitive Surgical's proprietaryEndoWrist™ articulating instruments, which are modeled after the humanwrist so that when added to the motions of the robot arm holding thesurgical instrument, they allow a full six degrees of freedom of motion,which is comparable to the natural motions of open surgery.

The da Vinci° surgeon's console has a high-resolution stereoscopic videodisplay with two progressive scan cathode ray tubes (“CRTs”). The systemoffers higher fidelity than polarization, shutter eyeglass, or othertechniques. Each eye views a separate CRT presenting the left or righteye perspective, through an objective lens and a series of mirrors. Thesurgeon sits comfortably and looks into this display throughout surgery,making it an ideal place for the surgeon to display and manipulate 3-Dintraoperative imagery.

A stereoscopic endoscope is positioned near a surgical site to captureleft and right views for display on the stereoscopic video display. Whenan instrument is outside a viewing area on the display, however, thesurgeon may not know how far away or in which direction the instrumentis at the time. This makes it difficult for the surgeon to guide theinstrument to the surgical site. Also, it may be disconcerting to thesurgeon if the instrument unexpectedly appears in view. Even when aninstrument is within the viewing area of the display, the surgeon maynot know which instrument it is or which patient-side manipulator (e.g.,robotic arm on the patient-side cart) the instrument is associated with.This makes it difficult, for example, for the surgeon to instruct apatient side assistant to replace the instrument with another during asurgical procedure.

In order to locate an instrument which is outside of a viewing area onthe display, it may be necessary to move the endoscope until theinstrument appears in the viewing area. In this case, if the surgicalinstrument is being guided to the surgical site, the cameras' zoom andfocus controls may also require frequent adjustment, making the processtedious and time consuming for the surgeon. If it happens that theinstrument is in the camera field of view (“FOV”), but outside of theviewing area, because of a zoom-in adjustment to the view, then azoom-out adjustment may be performed so that the instrument is back inthe viewing area. Such a zoom-out, however, may be undesirable when adelicate surgical procedure is being performed that requires closescrutiny by the surgeon.

OBJECTS AND BRIEF SUMMARY

Accordingly, one object of various aspects of the present invention is amethod for indicating a tool position relative to images being displayedon a computer display screen when the tool is outside a viewing area ofthe screen.

Another object of various aspects of the present invention is a methodfor indicating a tool distance from images being displayed on a computerdisplay screen when the tool is outside a viewing area of the screen.

Another object of various aspects of the present invention is a methodfor indicating a tool orientation relative to images being displayed ona computer display screen when the tool is outside a viewing area of thescreen.

Another object of various aspects of the present invention is a methodfor indicating a tool position or orientation relative to images beingdisplayed on a computer display screen when the tool is occluded withina viewing area of the screen.

Still another object of various aspects of the present invention is amethod for indicating a tool identification on a computer display screenthat clearly identifies which patient-side manipulators are connected towhich surgical instruments, so as to improve surgeon performance andsurgeon-assistant communications.

These and additional objects are accomplished by the various aspects ofthe present invention, wherein the embodiments of the invention aresummarized by the claims that follow below.

In preferred embodiments of the method, apparatus and medical roboticsystem, the symbol provides information identifying the tool and/or itsassociated patient-side manipulator by an associated color or some othermeans, such as text or numeric information that is written on ordisplayed adjacent to the symbol. In the latter case, the textinformation may be continuously displayed on the computer displayscreen. Alternatively, it may only be displayed when a cursor is placedover the symbol or the symbol is clicked on using a pointing device.

Additional objects, features and advantages of the various aspects ofthe present invention will become apparent from the followingdescription of its preferred embodiment, which description should betaken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a top view of an operating room employing a roboticsurgical system utilizing aspects of the present invention.

FIG. 2 illustrates two tools positioned in the FOV of an endoscopecamera.

FIG. 3 illustrates one tool positioned in and one tool positioned out ofthe FOV of an endoscope camera.

FIG. 4 illustrates a first computer display screen resulting from amethod for indicating a tool's position when the tool is out of the FOVof an endoscope camera, utilizing aspects of the present invention.

FIG. 5 illustrates a second computer display screen resulting from amethod for indicating a tool's position when the tool is out of the FOVof an endoscope camera, utilizing aspects of the present invention.

FIG. 6 illustrates a third computer display screen resulting from amethod for indicating a tool's position when the tool is out of the FOVof an endoscope camera, utilizing aspects of the present invention.

FIG. 7 illustrates a fourth computer display screen resulting from amethod for indicating a tool's position when the tool is occluded in theFOV of an endoscope camera, utilizing aspects of the present invention.

FIG. 8 illustrates a flow diagram of a method for indicating a tool'sposition when the tool is outside of, or occluded in, the FOV of anendoscope camera, utilizing aspects of the present invention.

FIG. 9 illustrates left and right views of a point in an endoscopecamera reference frame as used in a robotic surgical system configuredto perform the method described in reference to FIG. 8 which utilizesaspects of the present invention.

FIGS. 10 and 11 respectively illustrate a full left camera view beingdisplayed on a left viewing area of a computer monitor and partial leftcamera view being displayed on a left viewing area of a computermonitor.

FIG. 12 illustrates a flow diagram of a method for identifying a tool ina camera view that may be used in the method described in reference toFIG. 8 which utilizes aspects of the present invention.

DETAILED DESCRIPTION

FIG. 1 illustrates, as an example, a top view of an operating roomemploying a robotic surgical system. The robotic surgical system in thiscase is a Minimally Invasive Robotic Surgical (MIRS) system 100including a Console (“C”) utilized by a Surgeon (“S”) while performing aminimally invasive diagnostic or surgical procedure, usually withassistance from one or more Assistants (“A”), on a Patient (“P”) who islying down on an Operating table (“O”).

The Console includes a 3-D monitor 104 for displaying an image of asurgical site to the Surgeon, one or more manipulatable mastermanipulators 108 and 109 (also referred to herein as “control devices”and “input devices”), and a processor 102. The control devices 108 and109 may include any one or more of a variety of input devices such asjoysticks, gloves, trigger-guns, hand-operated controllers, or the like.The processor 102 is a personal computer that is integrated into theConsole or positioned next to it.

The Surgeon performs a minimally invasive surgical procedure bymanipulating the control devices 108 and 109 so that the processor 102causes their respectively associated slave manipulators 128 and 129(also referred to herein as “robotic arms” and “patient-sidemanipulators”) to manipulate their respective removably coupled surgicalinstruments 138 and 139 (also referred to herein as “tools”)accordingly, while the Surgeon views the surgical site in 3-D, as it iscaptured by a stereoscopic endoscope 140 (having left and right camerasfor capturing left and right stereo views) and displayed on the Console3-D monitor 104.

Each of the tools 138 and 139, as well as the endoscope 140, ispreferably inserted through a cannula or other tool guide (not shown)into the Patient so as to extend down to the surgical site through acorresponding minimally invasive incision such as incision 166. Each ofthe robotic arms is conventionally formed of linkages, such as linkage162, which are coupled together and manipulated through motor controlledjoints, such as joint 163.

The number of surgical tools used at one time and consequently, thenumber of robotic arms being used in the system 100 will generallydepend on the diagnostic or surgical procedure and the space constraintswithin the operating room, among other factors. If it is necessary tochange one or more of the tools being used during a procedure, theSurgeon may instruct the Assistant to remove the tool no longer beingused from its robotic arm, and replace it with another tool 131 from aTray (“T”) in the operating room. To aid the Assistant in identifyingthe tool to be replaced, each of the robotic arms 122, 128 and 129 mayhave an identifying number or color indicator printed on it, such as onits setup joint.

Preferably, the monitor 104 is positioned near the Surgeon's hands sothat it will display a projected image that is oriented so that theSurgeon feels that he or she is actually looking directly down onto theoperating site. To that end, an image of the tools 138 and 139preferably appear to be located substantially where the Surgeon's handsare located. To do this, the processor 102 preferably changes theorientations of the control devices 108 and 109 so as to match theorientations of their associated tools 138 and 139 as seen by theendoscope 140.

The processor 102 performs various functions in the system 100. Oneimportant function that it performs is to translate and transfer themechanical motion of control devices 108 and 109 to their respectiverobotic arms 128 and 129 through control signals over bus 110 so thatthe Surgeon can effectively move and/or manipulate their respectivetools 138 and 139. Another important function is to implement a methodfor indicating positions of a tool when the tool is outside a cameracaptured view being displayed on the monitor 104, or occluded within thecamera captured view being displayed on the monitor 104, as describedherein. Still another important function is to implement a method forreadily identifying tools and/or their respective patient-sidemanipulators on the monitor 104 to facilitate Surgeon/Assistantcommunications.

Although described as a personal computer, it is to be appreciated thatthe processor 102 may be implemented in practice by any combination ofhardware, software and firmware. Also, its functions as described hereinmay be performed by one unit, or divided up among different components,each of which may be implemented in turn by any combination of hardware,software and firmware.

During the performance of a minimally surgical procedure, the tools 138and 139 are preferably kept within a viewing area 200 of the monitor 104(such as shown in FIG. 2) so that the Surgeon may see them on themonitor 104 and accordingly, use them during the procedure. When one ofthe tools 138 is outside the viewing area 200 of the monitor 104 (suchas shown in FIG. 3), however, the Surgeon will be unable to see thattool on the monitor 104 and consequently, will be unable to properly useit during the procedure. In addition, the Surgeon may have difficultymoving the out-of-view tool into the viewing area 200 of the monitor 104without any knowledge of where the out-of-view tool is currentlypositioned relative to the viewing area 200.

To indicate tool positions to the Surgeon for out-of-view or occludedtools, the processor 102 is configured with a Graphical User Interface(“GUI”) computer program which implements a method for indicating toolpositions on the monitor 104, as described in reference to FIG. 8.Before describing this aspect of the GUI, however, examples of outputgenerated by the GUI are illustrated and described in reference to FIGS.4-7.

In each of the FIGS. 4-7, the viewing area 300 of the monitor 104 maycorrespond to the FOV of the endoscope 140 (with proper scaling of theentire FOV) such as depicted in FIG. 10, or it may correspond to only aportion of the FOV of the endoscope 140 (with proper scalingcorresponding to a ZOOM-IN of images in the portion of the FOV displayedon the monitor 104) such as depicted in FIG. 11. Tools within theviewing area 300 are seen in bold line in the viewing area 300.Circumscribing the viewing area 300 is a boundary area 400, in which,non-clickable symbols or clickable icons (hereinafter cumulativelyreferred to as “symbols”) are positioned so as to indicate positions ofcorresponding tools.

The symbols also preferably provide information identifying theirrespective tools and/or associated patient-side manipulators. One waythey may do this is by their colors which may match color indicationsprinted on the patient-side manipulators, such as on their setup joints.For example, patient-side manipulators 122, 128 and 129 may be colorcoded respectively as red, green and yellow, and symbols correspondingto their attached tools also color coded in the same manner.Alternatively, number indicators and/or other identifying informationmay be displayed on or adjacent to the symbols which may match numbersprinted on the patient-side manipulators, such as on their setup joints.For example, patient-side manipulators 122, 128 and 129 may be numbered1, 2 and 3 respectively, and symbols corresponding to their attachedtools also numbered in the same manner. Where text information isprovided with the symbols, the text may be written on or displayedadjacent to the symbol. It may be continuously displayed on the computerdisplay screen, or only displayed when a cursor is placed over thesymbol or the symbol is clicked on using a pointing device.

Tools outside the viewing area 300 are seen in dotted line for thepurposes of explaining certain aspects of the method implemented by theGUI. It is to be appreciated that these dotted lined tools (or dottedline tool extensions) are not seen by the Surgeon on the monitor 104.Their relative positions with respect to the viewing area 300 in FIGS.4-7, however, correspond to their relative positions in or to the FOV ofthe endoscope 140 in the endoscope camera frame of reference.

Although the tools shown in FIGS. 4-7 appear as 2-D images, it is to beappreciated that this is not to be construed as a limitation, but ratheras a simplification for descriptive purposes only. Preferably, 3-Dimages are displayed in the viewing area 300. The symbols and inparticular, end effector or tool shaft orientation indicationssuperimposed on the symbols, may appear in 2-D or 3-D in the boundaryarea 400. Also, although the examples described herein refer to imagescaptured by the endoscope 140, it is to be appreciated that the variousaspects of the present invention are also applicable to images capturedby other types of imaging devices such as those using MRI, ultrasound,or other imaging modalities, which may be displayed in the viewing area300 of the monitor 104.

FIG. 4 illustrates, as a first example, a GUI generated screen that isdisplayed on the monitor 104, wherein a first symbol 410 is placed inthe boundary area 400 to indicate the position of the out-of-view tool138, and an orientation indicator 411 is superimposed on the symbol 410to indicate the current orientation of an end effector 215 of theout-of-view tool 138. An in-view tool 139 is shown partially extendinginto the viewing area 300 from a second symbol 420 in the boundary area400.

In this example, the position of the first symbol 410 is determined bythe intersection of a line 402 and the boundary area 400, wherein theline 402 extends from a reference point on the out-of-view tool 138 to acentral point 401 of the viewing area 300 of the monitor 104. Theposition of the second symbol 420 is determined by the intersection ofthe shaft 222 of the in-view tool 139 and the boundary area 400.

The distance that the out-of-view tool 138 is away from the viewing area300, may be indicated in a number of ways, such as by the size, color,brightness/intensity, blinking frequency, or oscillating frequency ofits symbol. Alternatively, the distance may be simply indicated bydisplaying a distance number (such as the distance in centimeters) overthe symbol. For example, when the tool is in-view, such as the tool 139,then its symbol may be a maximum size, such as the symbol 420 of thein-view tool 139. When the tool is out-of-view, however, such as thetool 138, then the size of its symbol may indicate the distance that theout-of-view tool is away from the viewing area 300 so that it getslarger as the tool moves closer to entering the viewing area 300.Alternatively, the color of the symbol may indicate distance using acolor spectrum, or the brightness/intensity of the symbol or theblinking frequency of the symbol may indicate distance by increasing asthe tool moves closer to entering the viewing area 300, or anoscillation frequency of the symbol about its nominal position mayreduce as the tool is brought closer to being in the viewing area 300.

FIG. 5 illustrates, as a second example, a GUI generated screen that isdisplayed on the monitor 104, wherein a first symbol 510 is placed inthe boundary area 400 to indicate the position of the out-of-view tool138, and an orientation indicator 511 is superimposed on the symbol 510to indicate the current orientation of a shaft 217 of the out-of-viewtool 138.

In this example, the position of the first symbol 510 is determined bythe intersection of a line 502 and the boundary area 400, wherein theline 502 extends along an axis of the shaft 217. The distance that theout-of-view tool 138 is away from the viewing area 300, may be indicatedin the same manner as described above in reference to FIG. 4.

FIG. 6 illustrates, as a third example, a GUI generated screen that isdisplayed on the monitor 104, wherein a first symbol 610 is placed inthe boundary area 400 to indicate the position of the out-of-view tool138, and an orientation indicator 611 is superimposed on the symbol 610to indicate the current orientation of a shaft 217 of the out-of-viewtool 138.

In this example, the position of the first symbol 610 is determined bythe intersection of a trajectory 602 and the boundary area 400, whereinthe trajectory 602 is defined by the path of a reference point on theout-of-view tool 138 as it moves in the endoscope camera referenceframe. In this way the symbol 610 is placed in the boundary area 400where the tool will first appear in the viewing area 300 if it continuesalong its current trajectory (or, if it is moving away from the viewingarea 300, where it would appear if the trajectory were reversed). Forexample, if only two points in time are used to determine thetrajectory, as the tool 138 moves from a first location at time t1 to asecond location at time t2, the path of the reference point isrepresented by a line extending through the two points. If time t2 isthe current time and time t1 a prior time, then the current orientationof the shaft 217 is indicated by the orientation indicator 611. By usingmore than two points in time to define the trajectory of the out-of-viewtool 138, the trajectory may take on more sophisticated curves. Thedistance that the out-of-view tool 138 is away from the viewing area300, may be indicated in the same manner as described above in referenceto FIG. 4.

FIG. 7 illustrates, as a fourth example, a GUI generated screen that isdisplayed on the monitor 104, wherein both tools 138 and 139 arepositioned so as to be within the viewing area 300, but the end effectorof the tool 138 is occluded by an object 700. In this case, since eachof the tools is in the viewing area 300, their respective symbols 710and 420 are at maximum size. Although the end effector of the tool 138is occluded by the object 700, a ghost image 711 (e.g., a computermodel) of the end effector is shown at the proper position andorientation over the object 700. If the ghost image 711 is toodistracting, then an outline of the end effector may be used instead, aseither a programmed or surgeon selected option.

As previously described, the symbols 420, 410, 510, 610, and 710 may benon-clickable symbols or clickable icons. In the former case, if theSurgeon passes the cursor of a pointing device such as a mouse over thenon-clickable symbol, additional information about the associated toolmay be provided. In the latter case, if the Surgeon clicks on theclickable icon using the pointing device, additional information aboutthe associated tool may be provided. The additional information ineither case is information that is in addition to that identifying itsassociated patient-side manipulator, which may be indicated by its coloror a number that is always displayed on or adjacent to the symbol.Examples of such additional information may include identification ofthe tool's type and its associated master manipulator. The additionalinformation may be provided in a separate window such as apicture-in-picture, or it may be provided as text adjacent to, orsuperimposed over, the symbol. When the separate window is provided, theadditional information may further include a zoomed out, computergenerated view of the surgical site including the FOV of the endoscope140 and computer generated models of all tools outside of it.

Although shown as circles, the symbols 420, 410, 510, 610, and 710 maybe displayed in any one or more of many different shapes. For example,when the tool is positioned so as to be viewed inside the viewing area300, then the symbol may take the form of a computer model of the toolshaft so that a ghost shaft is displayed in the boundary area 400. Onthe other hand, when the tool is positioned so as to be outside of theviewing area 300, then the symbol may take the form of a computer modelof the distal end of the tool so that a ghost end effector is displayedin the boundary area 400. As the tool moves from outside of the viewingarea 300 into the viewing area 300, the symbol would then seamlesslychange from the ghost end effector to the ghost shaft, and vice versawhen the tool moves from inside of the viewing area 300 to outside ofthe viewing area 300. The orientation of the ghost shaft or ghost endeffector, as the case may be, would preferably match that of the actualtool. When the tool is outside of the viewing area 300, the size of theghost end effector may indicate its distance from the viewing area 300,as previously described for the symbols Likewise, in order to identifythe tool and/or its patient-side manipulator, the ghost shaft or ghostend effector, as the case may be, may be color coded or numericallynumbered as previously described for the symbols.

FIG. 8 illustrates, as an example, a flow diagram of a method forindicating a tool's position and identification on the monitor 104. Themethod is preferably performed for each tool by a GUI executed in theprocessing unit 102. In 801, the position and orientation of a tool aredetermined in the reference frame of an imaging device whose capturedimages are being displayed on the monitor 104. Although for the purposesof this example the images are described as being captured by the stereocameras of the endoscope 140, it is to be appreciated that imagescaptured by other imaging devices using other imaging modalities mayalso be used with the method. Also for the purposes of this example, thefull FOV of the cameras is assumed to be displayed in viewing area 300,such as depicted in FIG. 10. Therefore, in such case, the position andorientation of the tool may not be determinable using conventionalimaging techniques when the tool is outside the FOV of the cameras.

Consequently, the tool position and orientation (also referred to hereinas the “tool state”) are first estimated in a tool reference frame byreceiving information from joint sensors in the tool's robotic arm, andapplying the information to kinematics of the robotic arm. Because thetool state in this case is primarily determined from the robotic armkinematics, it can be readily determined even though the tool is outsidethe FOV of the endoscope 140 or occluded in the FOV of the endoscope140.

The estimated tool state is then translated into the camera referenceframe, and corrected using a previously determined error transform. Theerror transform may be determined from a difference between the toolstate determined using its robotic arm kinematics and a tool statedetermined using video image processing. The error transform may befirst determined with a pre-operative calibration step, and periodicallyupdated when the tool is in the FOV of the endoscope 140 during aminimally invasive surgical procedure.

If only a portion of the FOV of the cameras is displayed in viewing area300 of the monitor 104, however, such as depicted by area 1101 in FIG.11, then it may still be possible to use conventional imaging techniquesto determine the tool position if the tool is in a portion of the FOV ofthe cameras that is not being displayed in the viewing area 300 of themonitor 104, such as depicted by the area 1102 in FIG. 11. Note that inboth FIGS. 10 and 11, only the left camera view I1 is shown. It is to beappreciated, however, that for a 3-D display, a corresponding rightcamera view I2 is also necessary as described, for example, in referenceto FIG. 9, but is not being shown herein to simplify the description.

Additional details for determining tool positions and orientations, andin particular, for performing tool tracking are described, for example,in commonly owned U.S. application Ser. No. 11/130,471 entitled “Methodsand Systems for Performing 3-D Tool Tracking by Fusion of Sensor and/orCamera derived Data during Minimally Invasive Robotic Surgery,” file May16, 2005, which is incorporated herein by this reference.

In 802, a determination is made whether the position of the tool iswithin the viewing area 300 of the monitor 104, which is equivalent inthis example to determining whether the tool is within the FOV of theendoscope 140. This latter determination may be performed using epipolargeometry. Referring to FIG. 9, for example, the endoscope 140 includestwo cameras, C1 and C2, separated by a baseline distance “b”, and havingimage planes, I1 and I2, defined at the focal length “f” of the cameras.The image planes, I1 and I2, are warped using a conventional stereorectification algorithm to remove the effects of differing internal andexternal camera geometries.

A point P in the camera reference frame is projected onto the imageplanes, I1 and I2, at image points, P1 and P2, by an epipolar planecontaining the point P, the two optical centers of the cameras, C1 andC2, and the image points, P1 and P2. The position of the point P maythen be determined in the camera reference frame using known values forthe baseline distance “b” and focal length “f”, and a disparity “d”calculated from the distances of the image points, P1 and P2, from theirrespective image plane center points (i.e., at the intersections of thex-axis with the y1 and y2 axes).

Thus, in order for a tool to be in the FOV of the endoscope 140, atleast one point on the tool must be projected onto at least one of thetwo image planes, I1 and I2. Although it may be possible to estimate aposition of a point on the tool that is projected onto only one of thetwo image planes, I1 and I2, using disparity information calculated fornearby points, for example, preferably the point on the tool would beprojected onto both of the two image planes, I1 and I2, so that adisparity value may be calculated for the point and consequently, itsdepth can be determined directly. Also, although the tool maytechnically be in the FOV of the endoscope 140 if only one point of thetool is in it, for practical reasons, a sufficient number of points arepreferably required so that the tool is visually identifiable in themonitor 104 by the Surgeon.

Now, if the position of the tool is determined in 802 to be outside theviewing area 300 of the monitor 104, then in 803, a position for asymbol in the boundary area 400 circumscribing the viewing area 300 isdetermined such that the position of the symbol indicates the tool'sposition relative to the viewing area 300. Examples of suchdetermination have been previously described in reference to FIGS. 4-6.After determining the position of the symbol in the boundary area 400,in 804, the symbol is then displayed in the boundary area 400 at itsdetermined position. In addition, an orientation indicator may besuperimposed on the symbol and other tool and/or its robotic armidentifying information provided as described in reference to FIGS. 4-6.The method then repeats for another processing interval by going back to801.

On the other hand, if the position of the tool is determined in 802 tobe within the viewing area 300 of the monitor 104, then in 805, anattempt is made to identify the tool in the FOV of the endoscope 140.Referring to FIG. 12 as one example for performing this task, in 1201, a3-D computer model of the tool is generated. This is generally a onetime, pre-operative process. In 1202, the 3-D computer model of the toolis positioned and oriented according to the tool state determined in801. In 1203, right and left 2-D outlines of the computer model of thetool are generated by projecting an outline of the 3-D computer model ofthe tool onto the left and right image planes, I1 and 1I2, of the leftand right cameras, C1 and C2, of the endoscope 140. In 1204, the 2-Doutline of the computer model of the tool that was generated in 1203 forthe left image plane I1 is cross-correlated with a left camera viewcaptured by the left camera C1, and/or the 2-D outline of the computermodel of the tool that was generated in 1203 for the right image planeI2 is cross-correlated with a right camera view captured by the rightcamera C2.

In 806, a determination is then made whether the tool has beenidentified in the FOV of the endoscope 140 by, for example, determiningwhether a cross-correlation value calculated in 1204 meets or exceeds athreshold value for one or both of the left and right camera views. Ifthe result of 806 is a YES, then the tool has been identified in theright and/or left camera view. The method then goes to 803 to determinethe symbol position in the boundary area 400, which in this case may besimply determined by the intersection of the tool shaft with theboundary area 400. The method then proceeds to 804 to display the symbolin the determined position in the boundary area 400, and then to 801 torepeat the method for another processing period.

If the result of 806 is a NO, however, then the tool is presumablyoccluded by another object. In that case, in 807, the 2-D outline of thecomputer model of the tool that was generated by projecting the 3-Dcomputer model of the tool into the left image plane I1 is superimposedon the left camera view captured by the left camera C1, and the 2-Doutline of the computer model of the tool that was generated byprojecting the 3-D computer model of the tool into the right image planeI2 is superimposed on the right camera view captured by the right cameraC2. As a result, a 3-D outline of the computer model of the tool isdisplayed in the viewing area 300 of the monitor 104 superimposed overthe occluding object. Alternatively, the full 3-D computer model of thetool may be displayed as a ghost tool rather than just its outline overthe occluding object by superimposing appropriate left and right imagesof the 3-D computer model on the left and right camera views captured bythe cameras C1 and C2 of the endoscope 140.

The method then goes to 803 to determine the symbol position in theboundary area 400, which in this case may be simply determined by theintersection of the tool shaft with the boundary area 400. The methodthen proceeds to 804 to display the symbol in the determined position inthe boundary area 400, and then to 801 to repeat the method for anotherprocessing period.

Although the various aspects of the present invention have beendescribed with respect to a preferred embodiment, it will be understoodthat the invention is entitled to full protection within the full scopeof the appended claims.

What is claimed is: 1-27. (canceled)
 28. An apparatus comprising: amemory device non-transitorily storing program instructions; and aprocessor coupled to a display device, an image capture device, and thememory device, the processor configured to execute the programinstructions to: determine a position of a tool in a reference frame ofthe image capture device; determine a position to display anon-depictive symbol, which is non-depictive of the tool, in a boundaryarea circumscribing a viewing area on the display device, so as toindicate the position of the tool, while the viewing area is displayingan image captured by the image capture device; and cause thenon-depictive symbol to be displayed at the determined position in theboundary area.
 29. The apparatus according to claim 28, wherein theprocessor is configured to execute the program instructions to: causethe non-depictive symbol to be displayed at the determined position inthe boundary area so that the non-depictive symbol provides informationidentifying one of: the tool, and a manipulator used for moving thetool.
 30. (canceled)
 31. The apparatus according to claim 29, whereinthe processor is configured to execute the program instructions to:cause the non-depictive symbol to be displayed at the determinedposition in the boundary area so that the non-depictive symbol is markedwith one of: a color that is uniquely associated with the manipulator,and a number that is uniquely associated with the manipulator. 32.(canceled)
 33. (canceled)
 34. The apparatus according to claim 29,wherein the processor is configured to execute the program instructionsto: determine the position of the tool by using at least one of:kinematics for the manipulator, and an image identification technique.35. (canceled)
 36. The apparatus according to claim 28, wherein theprocessor is configured to execute the program instructions to:determine the position to display the non-depictive symbol in theboundary area by determining where a line passing through an axis, whichextends along a length of a shaft of the tool, enters a field of view ofthe image capture device.
 37. (canceled)
 38. The apparatus according toclaim 37, wherein the processor is configured to execute the programinstructions to: determine the position of the non-depictive symbol inthe boundary area by determining where a line extending from theposition of the tool to a reference point, which is in the field of viewof the image capture device, enters the field of view.
 39. The apparatusaccording to claim 28, wherein the processor is configured to executethe program instructions to: determine the position of the non-depictivesymbol in the boundary area by: determining a trajectory of the toolfrom current and past positions of the tool, and determining where anextrapolation of the trajectory enters the field of view of the imagecapture device.
 40. The apparatus according to claim 28, wherein theprocessor is configured to execute the program instructions to: causethe non-depictive symbol to be displayed at the determined position inthe boundary area by: determining a distance of a position of areference point on the tool from a position of a reference point in thefield of view of the image capture device, and causing the non-depictivesymbol to be displayed so that its size indicates the distance.
 41. Theapparatus according to claim 28, wherein the processor is configured toexecute the program instructions to: cause the non-depictive symbol tobe displayed at the determined position in the boundary area by:determining a distance of a position of a reference point on the toolfrom a position of a reference point in the field of view of the camera,and causing the non-depictive symbol to be displayed so that its colorindicates the distance.
 42. The apparatus according to claim 41, whereinthe processor is configured to execute the program instructions to:cause the non-depictive symbol to be displayed in the boundary area sothat the intensity of the color of the non-depictive symbol indicatesthe distance.
 43. The apparatus according to claim 41, wherein theprocessor is configured to execute the program instructions to: causethe non-depictive symbol to be displayed in the boundary area so thatthe color of the non-depictive symbol relative to a color spectrumindicates the distance.
 44. The apparatus according to claim 28, whereinthe processor is configured to execute the program instructions to:cause the non-depictive symbol to be displayed at the determinedposition in the boundary area by: determining a distance of a positionof a reference point on the tool from a position of a reference point inthe field of view of the camera, and causing the non-depictive symbol tobe displayed so that a frequency of blinking of the non-depictive symbolindicates the distance.
 45. The apparatus according to claim 28, whereinthe processor is configured to execute the program instructions to:cause the non-depictive symbol to be displayed at the determinedposition in the boundary area by: determining a distance of a positionof a reference point on the tool from a position of a reference point inthe field of view of the camera, and causing the non-depictive symbol tobe displayed so that a frequency of oscillation of the non-depictivesymbol about the determined position in the boundary area indicates thedistance.
 46. The apparatus according to claim 28, wherein the processoris configured to execute the program instructions to: cause thenon-depictive symbol to be displayed at the determined position in theboundary area by: determining a distance of a position of a referencepoint on the tool from a position of a reference point in the field ofview of the camera, and causing the non-depictive symbol to be displayedso that the distance is indicated by overlaying a distance number overthe non-depictive symbol.
 47. The apparatus according to claim 28,wherein the processor is configured to execute the program instructionsto: determine the position of the tool by determining a position and anorientation of an end effector of the tool, and cause the non-depictivesymbol to be displayed at the determined position in the boundary areaby causing an orientation indicator to be displayed over thenon-depictive symbol such that the orientation indicator is oriented soas to indicate the orientation of the end effector.
 48. The apparatusaccording to claim 28, wherein the processor is configured to executethe program instructions to: determine an orientation of an axis thatextends along a length of a shaft of the tool, and cause thenon-depictive symbol to be displayed at the determined position in theboundary area by causing an orientation indicator to be displayed overthe non-depictive symbol such that the orientation indicator is orientedso as to indicate the orientation of the axis.
 49. (canceled) 50.(canceled)
 51. An apparatus comprising: a memory device non-transitorilystoring program instructions; and a processor is configured to executethe program instructions to: cause images captured by an image capturedevice to be displayed in a viewing area on a display device; determinea position of a tool, which is within, but occluded in a field of viewof the image capture device, by determining a position of the tool in areference frame of the image capture device by using kinematics ofmanipulator used for moving the tool; and cause a ghost tool to bedisplayed in the viewing area where the tool is occluded by using thedetermined position of the tool in the reference frame of the imagecapture device.
 52. The apparatus according to claim 51, wherein theviewing area includes stereoscopic right and left two-dimensional views,and wherein the processor is configured to further execute the programinstructions to: determine that the tool is occluded in the field ofview of the image capture device by: generating a three-dimensionalcomputer model of the tool; positioning and orientating the computermodel to coincide with the current position and orientation of the toolin the reference frame of the computer display screen; generating atwo-dimensional outline of the computer model that projects onto aselected one of the right and left two-dimensional views; andcross-correlating the two-dimensional outline of the computer model withthe selected one of the right and left two-dimensional views.
 53. Theapparatus according to claim 52, wherein the processor is configured toexecute the program instructions to: cause the ghost tool to bedisplayed in the viewing area by causing at least the two-dimensionaloutline of the computer model of the tool to be displayed as an overlayon the selected one of the right and left two-dimensional views as it isbeing displayed on the computer screen.
 54. The apparatus according toclaim 52, wherein the processor is configured to execute the programinstructions to: cause the ghost tool to be displayed in the viewingarea by causing right and left two-dimensional ghost images of the toolto be overlaid in corresponding of the right and left two-dimensionalviews, so as to appear as a three-dimensional ghost image of the tool atthe position and orientation of the tool in the reference frame of thecomputer display screen. 55-76. (canceled)